Project 2 Board game object trackingΒΆ
- Krzysztof Weber 151934
- Adam Mielniczuk 151928
The game chosen by us is chess. The aim of the first milestone was to detect two game peaces. We were able to detect a chessboard and all the peaces. There are still some difficulties with the object detection but we are still working on it.
All functions used in latter examples are defined in utils.py file. It is to make the report more readable.
from utils import *
Data preparation.ΒΆ
We have prepared 9 recordings of 9 chess games. The are in three categories:
- Easy - Static camera, good lighting, hand covering parts of the board
- Medium - Static camera, shady lighting, hand covering parts of the board
- Hard - Shaky camera movements, shady lighting, hand covering parts of the board
Board detectionΒΆ
chess = video_from_path('recordings\Easy1.mp4')
Video loaded 912 720 30.00030000300003
frame = get_first_frame(chess)
Finding lines is made using HoughLinesP function along with canny edge detection. The lines are then represented with slopes and shifts defined in the form of y = ax + b and x = ay + b due to the fact that many lines are perfectly or almost perfectly vertical or horizontal.
lines = find_lines(frame)
Here we are looking for the squares on the chessboard. It is done by using the findContours function from cv2 on the threshold image. The we then calculate the convex hull of the contour to get rid of any distortions. If the contour has 4 edges and the lengths of the opposite edges are similar, then the contour is considered a chessboard square. Information about the chessboard squares: their with, height and the slopes of the edges will be used in chessboard grid detection.
gap_lengths, new_lines = find_square_parameters(frame)
lines.extend(new_lines)
Based on the square parameters we try to find a subset of 18 lines representing a grid. Lines should be in the similar distances from each other as the length of the square edges and their slopes should be similar to square slopes.
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.1)
Finally based on the chosen lines and their intersections we can define a chessboard grid with each cell being its own contour.
square_contours = find_square_contours(frame, chessboard_lines)
Chessboard tracking an figure detectionΒΆ
As on the hard videos the image is shaky the position of the chessboard must be tracked. To achieve this we find a bounding box by identifying bottom-left and top-right corners of the chessboard grid. The chessboard is then tracked by the MIL tracker. The grid is adjusted appropriately to the tracked bounding box.
The figures are found using a cv2 function goodFeaturesToTrack not to include chessboard corners we filer out those points that are close to grid lines.
Processing steps for each subsetΒΆ
EasyΒΆ
Easy1ΒΆ
chess = video_from_path('recordings\Easy1.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.1, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)
Video loaded 912 720 30.00030000300003
write_and_detect_chessboard(chess, square_contours, 'Easy1')
Easy2ΒΆ
chess = video_from_path('recordings\Easy2.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.18, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)
Video loaded 920 720 30.00030000300003
write_and_detect_chessboard(chess, square_contours, 'Easy2')
Easy3ΒΆ
chess = video_from_path('recordings\Easy3.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.15, if_display=False)
chessboard_lines = [chessboard_lines[0], chessboard_lines[2]]
square_contours = find_square_contours(frame, chessboard_lines)
Video loaded 876 720 30.00030000300003
write_and_detect_chessboard(chess, square_contours, 'Easy3')
MediumΒΆ
Medium 1ΒΆ
chess = video_from_path('recordings\Medium1.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.15, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)
Video loaded 866 720 30.00030000300003
write_and_detect_chessboard(chess, square_contours, 'Medium1')
Medium 2ΒΆ
chess = video_from_path('recordings\Medium2.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.15, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)
Video loaded 884 720 30.00030000300003
write_and_detect_chessboard(chess, square_contours, 'Medium2')
Medium 3ΒΆ
chess = video_from_path('recordings\Medium3.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.15, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)
Video loaded 876 720 30.00030000300003
write_and_detect_chessboard(chess, square_contours, 'Medium2')
HardΒΆ
Hard1ΒΆ
chess = video_from_path('recordings\Hard1.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
lines.extend(new_lines)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.2, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)
Video loaded 816 720 30.00030000300003
write_and_detect_chessboard(chess, square_contours, 'Hard1')
Hard2ΒΆ
from utils import *
chess = video_from_path('recordings\Hard2.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.2, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)
Video loaded 798 720 30.00030000300003
write_and_detect_chessboard(chess, square_contours, 'Hard2')
Hard3ΒΆ
chess = video_from_path('recordings\Hard3.mp4')
frame = get_first_frame(chess, if_display=False)
lines = find_lines(frame, if_display=False)
gap_lengths, new_lines = find_square_parameters(frame, if_display=False)
chessboard_lines = find_chessboard_lines(frame, lines, gap_lengths, aim_distance_correction=1.2, if_display=False)
square_contours = find_square_contours(frame, chessboard_lines)
Video loaded 720 752 30.00030000300003
write_and_detect_chessboard(chess, square_contours, 'Hard3')
Exemplary framesΒΆ
EasyΒΆ
MediumΒΆ
HardΒΆ
Problems encountered so far.ΒΆ
For some of the videos the reflection of the light on the chessboard makes the point detection algorithm find many false positives in that region. We are trying to solve that problem with including more measures to the figure tracking process: histogram, brightness.
For some of the video frames. The tracking is not perfect thus resulting in slight problems in figure recognition.